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Tutorial  Objectives 


Present  an  SoS  Architecture  Engagement  comprising  the  Mission 
Thread  Workshop,  the  Architecture  Challenge  Workshop,  the  SoS 
Architecture  Evaluation  and  the  System  and  Software  Architecture 
Evaluation  methods  in  the  context  of  a  DoD  mission-critical  SoS 
example 

Gain  an  understanding  of  the  benefits  of  applying  these  methods, 
including  the  points  in  the  acquisition  and  development  life  cycles 
where  each  method  provides  the  most  leverage 

Learn  how  to  identify  key  stakeholders  that  are  needed  to  make  the 
methods  successful 

Understand  how  the  results  of  these  engagements  can  be  applied 
within  programs  and  organizations  to  reduce  cost  and  risk  and 
improve  program  success 
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Outline 


Introduction 

SoS  Architecture  Engagement  Overview 
Mission  Thread  Workshop 

Methods/Activities  Superimposed  Over  DoD  SoS  Life-Cycle 

Architecture  Challenges  Workshop 

Legacy  System  &  Software  Architecture  Evaluation 

SoS  Architecture  Evaluation 

Next  Steps 
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Problem 


Integration  and  operational  problems  arise  due  to  inconsistencies, 
ambiguities,  and  omissions  in  addressing  quality  attributes  between 
system  and  software  architectures.  This  is  further  exacerbated  in  an 
SoS. 

Example  quality  attributes:  predictability  in  performance, 
availability/reliability,  security,  usability,  testability,  safety, 
interoperability,  maintainability,  force  modularity,  spectrum 
management 

Functionality  and  capability  are  important,  but  the  architecture  must  be 
driven  by  the  quality  attributes.  Identifying  and  addressing  quality 
attributes  early  and  evaluating  the  architecture  to  identify  risks  is  key 
to  success. 

Architecture  plays  an  important  role  in  every  stage. 
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Common  Symptoms 


Failure  to  address  quality  attributes  (non-functional  requirements)  in  the 
architecture  early  will  inevitably  lead  to  symptoms  such  as  these: 

Operational 

•  Communication  bottlenecks  under  various  load  conditions  throughout  SoS 

•  Systems  that  hang  up  or  crash;  portions  that  need  rebooting  too  often 

•  Difficulty  synching  up  after  periods  of  disconnect  and  resume  operations 

•  Judgment  by  users  that  system  is  unusable  for  variety  of  reasons 

•  Database  access  sluggish  and  unpredictable 

•  Lack  of  stability  in  overload  conditions 

Developmental 

•  Integration  schedule  blown,  difficulty  identifying  root  causes  of  problems 

•  Proliferation  of  patches  and  workarounds  during  integration  and  test 

•  Integration  of  new  capabilities  taking  longer  than  expected,  triggering  breaking 
points  for  various  resources 

•  Significant  operational  problems  ensuing  despite  passage  of  integration  and  test 

•  Anticipated  reuse  benefits  not  being  realized 

These  symptoms  often  point  to  architectural  deficiencies. 
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The  Need  for  Augmented  Mission  Threads  in 
DoD  SoS  Architecture  Definition 


DoDAF  is  the  SoS  architecture  framework  for  the  DoD.  It  provides  a 
good  set  of  architectural  views  for  an  SoS  architecture.  However,  it 
inadequately  addresses  cross-cutting  quality  attribute  considerations. 

System  use  cases  focus  on  a  functional  slice  of  the  system. 


More  than  DoDAF  and  system  use  cases  are  needed  to  ensure  that 
the  SoS  architecture  satisfies  its  cross-cutting  quality  attribute  needs. 


SoS  end-to-end  mission  threads  augmented  with  quality  attribute 
considerations  are  needed  to  help  define  the  SoS  Architecture 
precepts  and  guidelines,  and  then  later  evaluate  the  SoS  architecture. 
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Definitions  - 1 


A  System  of  Systems  is  “a  set  or  arrangement  of  systems  that  results 
when  independent  and  useful  systems  are  integrated  into  a  larger 
system  that  delivers  unique  capabilities.”  [OSD  Systems  Engineering 
Guide  for  Systems  of  Systems,  August  2008] 


OSD  SE  Guide  defines  four  types  of  SoSs: 

Directed 

Acknowledged 

Collaborative 

Virtual 

The  tutorial  will  be  addressing  Directed  and  Acknowledged  SoSs 
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Definitions  -  2 


Directed.  Directed  SoS  are  those  in  which  the  integrated  system-of-systems  is  built  and  managed  to 
fulfill  specific  purposes.  It  is  centrally  managed  during  long-term  operation  to  continue  to  fulfill 
those  purposes  as  well  as  any  new  ones  the  system  owners  might  wish  to  address.  The 
component  systems  maintain  an  ability  to  operate  independently,  but  their  normal  operational 
mode  is  subordinated  to  the  central  managed  purpose. 

Acknowledged.  Acknowledged  SoS  have  recognized  objectives,  a  designated  manager,  and 
resources  for  the  SoS;  however,  the  constituent  systems  retain  their  independent  ownership, 
objectives,  funding,  and  development  and  sustainment  approaches.  Changes  in  the  systems  are 
based  on  collaboration  between  the  SoS  and  the  system. 

Collaborative.  In  collaborative  SoS  the  component  systems  interact  more  or  less  voluntarily  to  fulfill 
agreed  upon  central  purposes.  The  Internet  is  a  collaborative  system.  The  Internet  Engineering 
Task  Force  works  out  standards  but  has  no  power  to  enforce  them.  The  central  players 
collectively  decide  how  to  provide  or  deny  service,  thereby  providing  some  means  of  enforcing 
and  maintaining  standards. 

Virtual.  Virtual  SoS  lack  a  central  management  authority  and  a  centrally  agreed  upon  purpose  for  the 
system-of-systems.  Large-scale  behavior  emerges — and  may  be  desirable — but  this  type  of  SoS 
must  rely  upon  relatively  invisible  mechanisms  to  maintain  it. 
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Definitions  -  3 


An  Architecture  is  the  structure  of  components,  their  relationships,  and 
the  principles  and  guidelines  governing  their  design  evolution  over 
time  [IEEE  Std  610.12  and  DoDAF]. 


An  SoS  Architecture  is  the  structure  of  constituent  systems,  their 
relationships,  and  the  principles  and  guidelines  governing  their 
design  evolution  over  time. 


Need  to  elaborate  on  this  to  clarify. 
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Elaboration 


The  structure(s)  of  the  constituent  systems  include: 

—  Allocation  of  functionality  to  each  constituent  system 

—  End-to-end  activity  flows  and  communications,  including  operational, 
sustainment,  development,  and  deployment  activities. 

—  Externally  visible  properties  and  interfaces  of  the  constituent  systems, 
including  behaviors,  dependencies,  use  of  shared  resources,  etc. 

—  Relationship  among  organizational  entities  and  the  constituent  systems  at 
each  phase  of  the  SoS  lifecycle. 

—  Rationale  and  governance  policies,  for  example,  criteria  for  decisions 
about  constituent  system  inclusion,  continued  participation  and 
termination. 


Depending  on  the  type  of  SoS: 

—  the  point  at  which  the  structures  are  determined  and  by  whom  can  vary 

—  the  level  of  specificity  and  abstractions  can  vary 
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Definitions  -  4 


Warfare  Vignette:  A  description  of  the  geography,  own  force 
structure  and  mission,  strategies  and  tactics,  the  enemy  forces  and  their 
attack  strategies  and  tactics,  including  timing.  There  may  be  associated 
Measures  of  Performance  (MOP)  and  Measures  of  Effectiveness 
(MOE).  A  vignette  provides  context  for  one  or  more  mission  threads. 

Mission  Thread: 

A  sequence  of  end-to-end  activities  and  events  beginning  with  an 
opportunity  to  detect  a  threat  or  element  that  ought  to  be  attacked  and 
ending  with  a  commander’s  assessment  of  damage  after  an  attack. 
C4ISR  for  Future  Naval  Strike  (Operational) 

Sustainment:  A  sequence  of  activities  and  events  which  focus  on 
development,  deployment  and  maintenance. 
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Vignettes  Are  the  Starting  Point  -  Example 
Context 
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Vignettes  Are  the  Starting  Point  -  Example 
Wording 


Two  ships  (Alpha  and  Beta)  are  assigned  to  integrated  air  and  missile 
defense  (IAMD)  to  protect  a  fleet  containing  two  high-value  assets 
(HVA).  A  surveillance  aircraft  SA  and  4  UAVs  are  assigned  to  the  fleet 
and  controlled  by  the  ships.  Two  UAVs  flying  as  a  constellation  can 
provide  fire-control  quality  tracks  directly  to  the  two  ships.  A  three¬ 
pronged  attack  on  the  fleet  occurs: 

•  20  land-based  ballistic  missiles  from  the  east 

•  5  minutes  later  from  5  aircraft-launched  missiles  from  the  south 

•  3  minutes  later  from  7  submarine-launched  missiles  from  the  west. 

The  fleet  is  protected  with  no  battle  damage. 
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Mission  Threads  Flow  from  Vignettes  -  Example 
(Non-Augmented) 


1. 20  land-based  missiles  launched  -  X  minute  window 

2.  Satellite  detects  missiles  -  cues  CMDR 

3.  CMDR  executes  re-planning  -  reassigns  Alpha  and  Beta 

4.  Satellite  sends  track/target  data  -  before  they  cross  horizon 

5.  Ships’  radars  are  focused  on  horizon  crossing  points 

N  Engagement  cycle  is  started  on  each  ship 
N+1 .  Aircraft  are  detected  heading  for  fleet 
N+2.  SA  detects  missile  launches  -  tells  CMDR 
N+3.  CMDR  does  re-planning  -  UAVs  are  re-directed 
N+4.  FCQ  tracks  are  developed  from  UAV  inputs 
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SoS  Architecture  Engagement  -  Overview 
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SoS  Architecture  Quality  Attribute  Specification  and 

Evaluation  Approach 


•  Early  elicitation  of  quality  attribute  considerations 

•  Early  identification  and  addressing  of  architecture  challenges 

•  Early  identification  and  mitigation  of  architectural  risks 
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Identify  and  Address  Architectural  Challenges  -  Early 

Early  elicitation  of  quality  attribute  considerations 
Early  identification  and  mitigation  of  architectural  risks 


SoS  Business  /  Mission  Drivers 


Warfare  Vignettes 
Mission  Threads 


SoS  Architecture 


Plans  v 


Mission 

Thread 

Workshop 


1 1 

Architecture 

Challenge 

Workshop 

A  A 


Augmented  Mission  Threads 
SoS  Architecture  Challenges 


y 


SoS 

Architecture 

Evaluation 


SoS  Architecture  Risks 

Problematic  systems 
identified  with  the 
augmented  mission 
threads 


SoS  Architecture 
System  Architectures 


t 


SoS  and  System  Architecture(s)  Acquisition  /  Development 
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Legacy  System  Architecture  Evaluation  -  Early 

•  Early  elicitation  of  quality  attribute  considerations 


•  Early  identification  and  mitigation  of  architectural  risks 
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Mission  Thread  Workshop 
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Mission  Thread  Workshop  (MTW)  Purpose 


The  MTW  augments  SoS  mission  threads  with  quality  attribute 
considerations  that  shape  the  SoS  architecture  and  identifies  SoS 
architectural  challenges,  as  early  in  the  SoS  development  cycle 
as  possible. 

The  mission  thread  augmentation  is  performed  with  inputs  from  key 
SoS  stakeholders  and  is  facilitated  by  the  SEI. 

The  augmented  mission  threads  and  challenges  are  used  to 
develop  the  SoS  architecture  and  then  later  to  evaluate  the  SoS 
architecture. 

There  will  be  a  series  of  MTWs  depending  on  scope,  scale,  and 
schedule  considerations. 
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MTW  sequence  planning/scheduling  and 
vignette  and  MT  development/selection 

Criteria  for  development/selection  of  vignettes  and  MTs 

•  Capability  Coverage 

New  requirements/capabilities 

•  Stressing  the  SoS 

•  constituent  systems,  communications,  etc 
New  integrated  existing  capabilities 

You  can  only  do  so  many  of  these...  make  them  count. 
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MTW  Inputs  - 1 


SoS  Business  and  Mission  Drivers  Presentation  (15  mins) 

•  A  representative  from  the  SoS  stakeholder  community  presents  the 
SoS  business  and/or  mission  drivers  including  the 
business/programmatic  context,  high-level  functional  requirements, 
high-level  constraints,  high-level  quality  attributes,  acquisition  strategy, 
etc. 


SoS  Architecture  Plans  Presentation  (30  mins) 

•  The  SoS  architect  presents  the  architecture  development  plans 
including  key  business/programmatic  requirements,  key  technical 
requirements  and  constraints  that  will  drive  architectural  decisions,  any 
relevant  existing  context  diagrams,  high-level  SoS  diagrams  and 
descriptions,  development  spirals  and  integration  schedule. 
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MTW  Inputs  -  2 


Vignettes 

•  A  description  of  the  geography,  own  force  structure  and  mission, 
strategies  and  tactics,  the  enemy  forces  and  their  attack  strategies  and 
tactics,  including  timing.  There  may  be  associated  Measures  of 
Performance  (MOP)  and  Measures  of  Effectiveness  (MOE). 

—  An  SoS  will  typically  support  multiple  vignettes,  i.e.  multiple  mission 
areas  such  as  Air  Defense,  Ballistic  Missile  Defense, 
Replenishment,  Mobility,  etc. 

—  Each  vignette  typically  supports  multiple  mission  threads 

Mission  Threads,  types: 

•  Operational  -  A  sequence  of  activities  and  events  beginning  with  an 
opportunity  to  detect  a  threat  or  element  that  ought  to  be  attacked  and 
ending  with  a  commander’s  assessment  of  damage  after  an  attack. 

•  Sustainment:  A  sequence  of  activities  and  events  which  focus  on 
development,  deployment  and  maintenance. 
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Preparation 


The  SoS  Program  Manager  develops  a  overview  presentation  on  the 
SoS  Mission  /  Business  Drivers  (see  SoS  Mission  /  Business  driver 
presentation  template). 

The  SoS  Architect  develops  an  overview  presentation  on  the  SoS 
Architecture  Plans  (see  SoS  Architecture  Plans  presentation 
template). 

The  SEI  meets  with  the  SoS  Architect  and  PM  to: 

•  Determine  if  the  vignettes  and  MTs  are  sufficient  to  proceed. 

•  Provide  feedback  on  the  two  presentations 

•  Reach  agreement  on  scope  and  series  of  MTWs 

•  Identify  Stakeholders 

•  Determine  logistics 
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Stakeholders  are  Key! 


When  developing  the  initial  set  of  vignettes  and  MTs,  it  is  critical  to 
associate  them  with  the  key  stakeholder  types  that  will  be 
necessary  to  participate  in  the  Workshops. 


There  may  be  groups  of  stakeholder  types  that  are  not  necessary  for 
specific  vignettes. 


Example  stakeholders:  (leads  in  the  following) 
Modeling  and  Simulations 
Integration  and  Test  Facility  (SIL) 
CONOPS,  DRM,  Operational  Analysts, 
SoS,  System  and  Software  Architects 
•  Legacy  System  Architects 
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Typical  MTW  Agenda 


08:00-08:15 

08:15-08:45 

08:45-09:00 

09:00-09:40 

09:40-09:55 

09:55-12:00 

12:00-13:00 

13:00-13:20 

13:20-15:00 

15:00-15:15 

15:15-15:45 

15:45-17:00 


Welcome/Introductions/Opening  Remarks  ( joint ) 

MTW  Overview  (SEI) 

Business  Drivers  and  Quality  Attributes  (Architect) 

OV-1  &  Vignettes  Overview  (Architect) 

Break 

Augmentation  of  1st  mission  thread  (SEI  facilitated) 

Lunch 

Review  OV-1  and  vignette  associated  with  2nd  mission  thread  (Architect) 
Augmentation  of  2nd  mission  thread  (SEI  facilitated) 

Break 

Review  OV-1  and  vignette  associated  with  3rd  mission  thread  (Architect) 
Augmentation  of  3rd  mission  thread  (SEI  facilitated) 
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Augmentation  Process  -  Per  Mission  Thread 


1)  For  each  event  in  the  mission  thread: 

•  Elicit  quality  attribute  considerations.  Capturing  any  engineering  issues,  assumptions, 
challenges,  additional  use  case  and  mission  threads  (with  QA  context  etc.) 

•  Capture  any  capability  and/or  mission  issues  that  arise. 

2)  Elicit  any  over-arching  quality  attribute  considerations 

•  Capturing  any  over-arching  assumptions,  engineering  issues,  challenges,  additional 
use  cases  and  mission  threads  (with  QA  context)  etc. 

3)  Capture  any  capability  and/or  mission  issues  that  arise. 

4)  Capture  any  MT  extensions  for  a  later  pass. 

Parking  Lot  -  for  organization,  programmatic,  non-technical  issues  that  arise  (will 
not  be  further  pursued  in  the  MTW). 


SEI  facilitates  and  scribes  using  a  pre-defined  MTW  template. 

Stakeholder  Inputs  are  Key. 
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Rules 


SEI  will  provide  the  facilitation  and  scribing. 

This  is  a  big  crowd:  side  conversations,  cell  calls,  etc.  will  not  be 
allowed  to  disrupt  the  meeting. 

Once  an  issue  is  identified  and  discussed,  we  will  not  allow  it  to  be  re¬ 
discussed.  It  will  be  noted  at  the  appropriate  place. 

Will  keep  the  discussions  within  scope. 

Will  not  get  into  the  details  of  potential  solutions  to  issues. 

Programmatic,  organizational,  and  other  non-technical  issues  will  be 
noted,  but  not  discussed  in  detail. 
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Example  MTW  Walk-Through 


At  this  point  in  the  tutorial  we  will  switch  to  the  MTW  template  which 
partially  filled  in.  We  will  walk  through  the  MTW  augmentation 
process  using  the  DoD  SoS  example. 

Starting  with  the  example  business  driver  and  architecture  plans 
presentations,  then  proceeding  to  the  example  mission  thread 
augmentation. 
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MTW  Presentation  Topics 


Scope  and  Stakeholders 
Business  Drivers 

•  Design  Precepts 

•  Engineering  Strategy 

Quality  Attributes 

•  Architecture  Quality  Attributes 

•  Technical  Model 
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Scope 


Interested  in  “Whole  Ship”  level  interactions  with  other  assets, 
invited  stakeholders 

•  Engagement  management,  Communications 

•  Missiles,  Radar,  UAV,  Helo 

•  Analysts,  Planner,  Survivability 

•  Modeling  and  simulation,  Test 

•  Programmatic 


Identify  missing  use  cases 

Identify  additional  engineering  analysis  tasks 
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Design  Precepts  1 


Life-cycle  costs 

•  Ease  of  component  removal  and  replacement  for  maintenance 
and  modernization 

•  Open  Architecture  COTS  solutions 

•  Effective  Resource  management  (power,  cooling,  inter¬ 
connectivity,  interface  controls,  weight,  and  volume) 
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Design  Precepts  2 


Government  led  Design  using  Integrated  Product  and  Process 
Development  with  Industry  and  Government  teams 

•  Key  IPTs  and  Working  Groups  Co-Chaired  by  Program  and 
Technical  leads 

•  Technical  Authority  applied  in  periodic  reviews  and  issue 
resolution 

•  All  Design  Characteristics  will  be  traceable  to  requirements 
references  through  Total  Ship  Systems  Engineering  Process 
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Engineering  Strategy 


Decouple  product  development  from  platform  development 
Re-use  (and  potentially  re-engineer)  existing  POR  products 
Strive  for  commonality  across  ship  classes 
Government  owned  architecture 
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Quality  Attributes 


End  User  Impact 

Interoperation  Impact 

Acquisition  Impact 

Performance 

Availability 

Reliability 

Maintainability 

Fault  Tolerance 

Survivability 

Safety 

Usability 

Mission  Flexibility 

Accuracy 

Supportability 

Interoperability 

Backward  Compatibility 
Network-Centricity 

Information  Exchange 
Information  Assurance 

Security 

Privacy 

Openness 

Reusability 

Affordability 

Testability 

Extensibility 

Scalability 

Adaptability 

Expandability 
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Example  Operational  View 


Software  Engineering  Institute  CarnegieMellon 


NDIASE  Tutorial  -2009 
Gagliardi,  Wood,  Morrow,  Klein 
©2009  Carnegie  Mellon  University 


38 


MTW  Outputs 


Individual  MTWs 

Augmented  Mission  Threads  (.doc,  using  MTW  template) 

•  Over-arching  quality  attribute  augmentations  for  the  mission  thread 

•  Capability  and  mission  augmentations  to  the  mission  thread 

•  Quality  attribute  augmentations  for  each  event  in  the  mission  thread 

•  Identified  mission/additional  use  cases  (with  context)  and  mission  threads 

Challenges  (.ppt  slides,  vetted  with  sponsor) 

•  Architectural,  capability  and  mission  challenges  derived  from  the  mission  thread  augmentations. 

•  The  MTW  team  will  roll  up  challenges  from  the  data  and  provide  an  out-brief  of  the  challenges. 

•  Mapped  to  contributing  augmented  mission  thread  steps 

•  These  are  vetted  and  updated  with  the  principals 

•  Identify  any  candidate  legacy  system  architecture  that  may  require  architecture  evaluation. 

•  Refer  to  the  example  MTW  template  here. 


SoS  Architectural  Challenges  (.ppt  slides,  vetted  with  sponsor) 

Report  upon  completion  of  series  of  MTWs: 

•  SoS  architectural  challenges  derived  and  rolled  up  from  the  mission  thread  augmentations; 
upon  completion  of  the  series  of  mission  thread  workshops  for  the  SoS. 

•  Meet  with  the  principals  to  “rack  and  stack”  challenges. 
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Examples  of  Rolled-up  Challenges 


Address  resource  management  issues  dealing  with  supporting 
the  number  of  missiles  and  radar  coverage 

Performance  timelines  and  deadlines  need  defined  and 
decomposed 

Engineering  studies/analyses  insufficient  in  area  of 
manning/automation 

Develop  a  better  understanding  of  external  interfaces  due  to 
the  impacts  they  will  have  on  the  system. 

Sensor  coordination  between  the  two  ships  and  the  UAVs 
needs  further  refinement. 
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MTW  Experiences  - 1 


Conducted  a  total  of  15  MTWs,  each  1  -2  day  meetings 

Plan  4  MTs  per  MTW,  but  expect  to  augment  3. 

Expect  25-30  stakeholders  to  want  to  participate  per  MTW.  Benefits 
from  strong  facilitation  and  independent  3rd  party  leadership. 

Clients  developed  very  good  first  pass  vignettes  and  MTs  after  initial 
introduction 

Criteria  for  MT  selection  include:  New  capability,  High  perceived  risk, 
proposal  differentiators,  etc. 

DoDAF  OV-Ts  were  sufficient  level  of  documentation  going  into  the 
MTWs 
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MTW  Experiences  -  2 


Most  of  the  time  taken  by  step  elaboration,  for  example 

•  Command  authority,  network  communications,  step  constraints 

•  Manned  vs  Automated,  timelines,  planning  considerations 

•  Availability  and  Survivability  considerations 

•  Readiness,  environmental  conditions,  start  up/shut  down 

•  Current  capabilities/extensions 

•  CONOPS  missing 

•  Assumptions 

Extensions 

•  Clients  built  some  initially 

•  Added  them  as  we  go  (to  sideline  discussions) 
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MTW  Experiences  -  3 


Quality  Attributes 

•  Timeline  often  built  into  thread  (weeks  to  seconds) 

•  Availability/  Degraded  Operation  /  Resource  Management  under¬ 
developed 

•  Focus  on  operational  MTs,  separate  MTW  for  development  and  support 

•  Over-arching  MT  pass  collects  much  of  the  QA  considerations 

•  Identified  additional  use  cases  and  MTs  (e.g.  survivability) 

Challenges 

•  Currently  doing  on  a  MT  basis 

•  Some  challenges  need  to  be  kicked  up  to  the  SoS  Architecture  level  to 
address.  Implies  a  SoS  Architecture  and  Guidelines  Document. 
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MTW  -  Initial  Results  - 1 


The  MTW  and  SoS  Arch  Evaluation  methods  adopted  by  the 
organization  and  required  in  their  architecture  development 
process 

Many  of  the  identified  challenges  drove  some  early  risk  mitigation 
activities  (e.g.  prototyping,  EDM,  white  papers). 

Many  new  use  cases  and  additional  mission  threads  identified.  The 
QA  considerations  will  be  included  in  the  use  cases. 

Excellent  vehicle  to  promote  communication  between  architects  and 
stakeholders. 

Capability  and  Mission  Challenges  were  identified  as  well  as 
Architectural  Challenges. 
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MTW  -  Initial  Results  -  2 


SoS  Architecture  and  Guidelines  document  is  needed.  Developed  a 
template  for  use  on  an  Army  program. 


Supports  programs’  DoDAF  architecture  development  efforts 


3rd  Party  facilitation  by  the  MTW  facilitators  enabled  the  leads  to  think 
about  and  participate  in  the  discussions  rather  than  trying  to 
lead/control  the  meetings 


Method  worked  for  non-software  elements,  as  well  as  software¬ 
intensive  elements 
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Methods/Activities  Superimposed  Over  DoD 
SoS  Life-Cycle 
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Material  Solution  Analysis  Phase 


Material  Solution  Analysis  Phase 


MOD 

XJ 


MS  A 

XJ 


Material  Solution  Analysis 
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Technology  Development  Phase 


XJ 


XJ 


AS 

SEP 

TEMP 


Technology  Development  Phase 
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Identify  and  Address  Architectural  Challenges  -  Early 

Early  elicitation  of  quality  attribute  considerations 
Early  identification  and  mitigation  of  architectural  risks 


SoS  Business  /  Mission  Drivers 


Warfare  Vignettes 
Mission  Threads 


SoS  Architecture 


Plans  v 


Mission 

Thread 

Workshop 


1 1 

Architecture 

Challenge 

Workshop 

A  A 


Augmented  Mission  Threads 
SoS  Architecture  Challenges 


y 


SoS 

Architecture 

Evaluation 


SoS  Architecture  Risks 

Problematic  systems 
identified  with  the 
augmented  mission 
threads 


SoS  Architecture 
System  Architectures 


t 


SoS  and  System  Architecture(s)  Acquisition  /  Development 
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Outline  of  ACW 


Purpose:  To  resolve  a  challenge  from  the  MTW 

Inputs:  Vignettes  /  thread  steps  that  contribute  to  the  ACW 

Preparation:  Preliminary  technical  analysis 

Processing:  Review  the  challenges  impact  and  develop 
aspects  of  the  challenge 

Outputs:  Plan  to  handle  challenge  aspects 
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Resource  Management-  Purpose 


To  describe  the  resource  management  issues  that  arose 
during  the  MTW,  and  organize  them  such  that  the  design 
can  resolve  these  issues. 
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Resource  Management-  Inputs 


Missiles  :  Steps  1  ,  8,  9,  10,  14,  15,  16,  20,  Ex2 
Radar  Coverage:  2,6,7,11,  Exl  (all) 
Communications:  Av2,  AV3,  AV4,  Av6,  Av7 
Degraded  Operations:  Av5 
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Resource  Management  -  Preparatory 


The  air  defense  (AD)  daily  planning  must  include 

•  The  radar  coverage  of:  SA,  4  UAVs,  own  radar 

•  Missiles  available  for  AD  and  their  status  (both  ships) 

The  AD  planning  to  handle  imminent  threats 

•  Assignment  of  incoming  missiles  to  Alpha  and  Beta  for 
engagement 

There  are  a  number  of  fault  conditions  that  impact  operations 

•  Lack  of  radar  coverage 

•  Communication  failures 

Degraded  modes  must  be  defined  clearly 
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Resource  Management-  Rack  and  Stack 


AD  Daily  Planning 

Low 

High 

Low 

4 

AD  Threat  Planning 

High 

High 

High 

1 

Fault  Conditions 

High 

High 

Low 

2 

Degraded  Modes 

Med. 

Low 

Low 

3 
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Resource  Management-  Processing 


Agenda 

Review  and  edit  the  input  material 

Review  and  edit  the  preparatory  material 

Determine  the  segments  impacted 

Define  the  interactions  between  these  segments  and  the 
interactions  with  external  actors 

Plan  the  design  steps 

•  White  papers,  prototyping,  experiments,  design,  etc. 
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Resource  Management  -  Outputs 


Perform  simulation  studies  to  determine  how  best  to  allocate 
resource  coverage  from  different  resources 

Write  a  white  paper  on  AD  engagement  assignment 

•  Current  approach  and  shortcomings 

•  Study  alternative  approaches 

•  Suggest  what  should  be  done 

Write  a  white  paper  on  failure  conditions  that  can  arise  and  the 
recovery  procedures  that  could  be  invoked 

A  first  pass  definition  of  degraded  operational  modes  was 
made  in  briefing  form 

Schedule  of  above  activities 
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Typical  Architectural  Challenge  Workshop 
Agenda 

08:00-08:15  Welcome/Introductions/Opening  Remarks  ( joint ) 

08:15-08:45  ACW  Overview  (SEI) 

08:45-09:40  OV-1 /Vignettes/Augmented  MT  Steps  Associated  with  Arch 
Challenge  (Architect) 

09:40-09:55  Break 

09:55-12:00  Review  and  develop  aspects  of  the  challenge  (SEI  facilitated) 
12:00-13:00  Lunch 

13:00-15:00  Review  and  discuss  the  architects’  architectural  approaches  (SEI 
Facilitated) 

15:00-15:15  Break 

15:15-17:00  Plan  the  design  steps  (SEI  facilitated) 
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Legacy  System  Architecture  Evaluation 
Using  ATAM 
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Legacy  System  Architecture  Evaluation  -  Early 

•  Early  elicitation  of  quality  attribute  considerations 


•  Early  identification  and  mitigation  of  architectural  risks 
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Is  a  System  ATAM  Variant  Appropriate  For 
Defensive  Engagement  System? 


Comments  from  augmented  mission  thread: 

The  Defensive  Engagement  System  may  not  be  able  to  support  the  deconfliction 
timeline  for  5  incoming  missiles. 

The  Defensive  Engagement  System  may  not  have  the  capability  to  acknowledge 
Beta’s  acceptance  of  its  assignment  of  2  missiles. 

Is  the  Defensive  Engagement  System  capable  of  sending  track  updates  to  the 
interceptor  missiles  that  Beta  had  launched  within  the  intercept  timeline? 

In  Phase  0,  the  System  ATAM  lead  meets  with  SoS  and  appropriate  system 
architects  to  discuss  what  is  in  and  out  of  scope  concerning  the  system 
under  analysis  and  if  appropriate  documentation  exists 

Agree  on  scenarios  based  upon  the  augmented  mission  thread,  with  the 
understanding  that  additional  scenarios  can  be  added  during  Phase  2  of 
the  System  ATAM 
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Conceptual  Flow  of  System  ATAM  Variant 


Business 

Drivers 


System  and 
Software 
Architecture 


Impacts 

Also  SoS  vs  System 
tradeoffs 


Augmented 
Mission  Threads  and 
Use  Cases  based  on  MTWs 


Quality 

Attributes 


Architectural 

Approaches 


Risk  Themes 


distilled 
into 


Scenarios 


Architectural 

Decisions 


Tradeoffs 


Sensitivity  Points 


Non-Risks 


Risks 
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Examples  of  Scenarios 


Scenarios  address  both  system  and  software  aspects 
Use  case  scenario 

The  Defensive  Engagement  System  (DES)  is  able  to  support  de- 
confliction  of  7  incoming  missiles  using  own-ship  and  external 
information  within  XX  seconds. 

Growth  scenario 

An  upgraded  DES  is  able  to  reduce  the  confliction  time  by  40%  of  7 
incoming  missiles  with  no  loss  of  existing  functionality. 

Exploratory  scenario 

The  DES  is  able  to  operate  at  up  to  80%  of  its  time  budget  for  de- 
con  f liction  of  7  incoming  missiles  with  8  coalition  UA  Vs  and  3  coalition 
helicopters  operating  in  its  vicinity. 
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ATAM  Phase  2  Specifics 


Stakeholders  will  consist  of: 

System  Architects  of  relevant,  associated  systems  to  system  under 
evaluation 

SoS  Architects  who  know  the  total  system  and  how  the  system  under 
evaluation  is  envisioned  to  fit  in 

Relevant  stakeholders  of  the  system  under  evaluation  in  the  areas  of 
requirements,  development,  T&E,  sustainment,  M&S 


ATAM  evaluators  will  look  to  identify/expose  potential  system  and 
software  architecture  risks,  with  the  help  of  the  stakeholders. 
Subject  matter  experts  may  be  used  on  the  evaluation  team,  if 
necessary. 
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Walk-through  of  a  scenario  derived  from 
augmented  MT 


The  Defensive  Engagement  System  (DES)  is  able  to  support  de- 
confliction  of  7  incoming  missiles  using  own-ship  and  external 
information  within  XX  seconds. 

System  architect  identifies  that  currently  DES  can  support  3  incoming 
missiles  with  25%  spare  capacity  given  the  existing  hardware.  The 
architect  also  states  that  the  system  has  a  monolithic  software 
architecture  which  is  tightly  coupled  to  the  hardware. 

The  architect  identifies  that  upgraded  hardware  is  available  for  the 
system  which  will  improve  performance,  but  the  software  will  need 
to  be  re-designed  to  support  it. 


DES  software  architecture  risk  identified  early  and  mitigations 

planned 


=  Software  Engineering  Institute  CarnegieMellon 


NDIASE  Tutorial  -2009 
Gagliardi,  Wood,  Morrow,  Klein 
©2009  Carnegie  Mellon  University 


66 


SoS  Architecture  Evaluation 
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SoS  Architecture  Evaluation  Purpose 


The  SoS  Architecture  Evaluations  identifies  SoS 
architectural  risks  by  probing  the  SoS  architecture,  using 
the  augmented  SoS  mission  threads  and  challenges,  to 
evaluate  the  SoS  architecture.  It  also  identifies  any 
problematic  systems  that  require  further  evaluation. 


There  will  be  a  series  of  SoS  Architecture  Evaluations 
depending  on  scope,  scale,  and  schedule  considerations. 
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Evaluation  Approach  - 1 


Similar  to  ATAM  in  some  ways 

•  Appropriate  architecture  documentation  required 

•  Stakeholders  required  throughout 

•  Architect(s)  walk  the  augmented  mission  thread  through  the 
SoS  architecture  with  evaluation  team  probing  for  risks,  non¬ 
risks,  etc. 

•  2  day  max  per  evaluation 

•  not  a  precise,  exhaustive  evaluation 

•  Risks  rolled  up  into  risk  themes 

•  Evaluation  team  required  throughout 

•  Scoping  is  critical 
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Evaluation  Approach  -  2 


Differs  from  ATAM  in  some  ways 

•  Use  existing  augmented  mission  threads  from  the  MTW 

•  Requires  execution  of  a  MTW  prior  to  evaluation 

•  Mission  threads  augmentation  nor  occurring  during  the  evaluation 

•  Identify  problematic  areas  for  more  focused  architecture 
evaluation 

•  Initial  preparation  requires  proper  scoping  and  development  of  a 
scheduled  series  of  SoS  Arch  Evals: 

•  Ensure  proper  stakeholder  representation;  balance  between  not 
wasting  anyone’s  time  versus  benefits  of  participation  and 
communication.  Depends  on: 

•  Mission  thread  “type”  -  operational,  sustainment 

•  Clustering  of  constituent  systems  per  mission  thread 

•  Constrained  by  time  it  takes  to  go  through  a  mission  thread  (1  per  day) 
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Evaluation  Approach  -  3 


Three  stages 

•  Preparation 

•  Execution 

•  Roll-up  and  Follow-up 
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Stage  1 :  Preparation  - 1 


Review  results  of  MTW,  noting  the  architectural  challenges  and  expected 
resolutions;  and  highlight  augmentations  that  require  further 
explanation 

Identify  the  mission  threads  for  the  SoS  Arch  Eval  with  the  SoS  architect 
•  Assume  that  only  1-2  mission  threads  can  be  evaluated  per  day  max. 

Develop  and  review  the  SoS  business/mission  drivers  and  the  SoS  and 
System/SW  architecture  presentations 

Review  SoS  and  system  architecture  documentation  for  sufficiency 

Identify  stakeholders  (some  to  assist  with  the  evaluation) 
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Stage  1 :  Preparation  -  2 


Develop  a  schedule  of  the  evaluations 

Set  up  logistics  and  send  out  read-ahead  with  invitations 

Walk-through  one  mission  thread  for  practice 

Identify  evaluation  team 

•  Lead,  Scribe,  3  Evaluators 

—  ATAM  evaluator  qualified 

•  Domain  SMEs  (e.g.  Communications,  sensors,  weapons,  platforms, 
warfare  experts) 

Evaluation  team  reviews  the  inputs  and  becomes  familiar  with  the  SoS 
Architecture  in  advance  of  the  evaluation 
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Stage  2:  Execution  - 1 


Note:  2  day  max  for  each  SoS  Arch  Eval 

•  Probably  will  only  get  through  2  mission  threads 

Presentations: 

•  SoS  Business/Mission  Driver  Presentation 

•  SoS  Architecture  Presentation 

•  Augmented  Mission  Threads  for  this  evaluation 

•  Architectural  Challenges  from  the  MTW 
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Stage  2:  Execution  -  2 


Analysis  for  each  architecture  challenge 

•  The  architect  describes  how  the  architecture  satisfies  each 
architecture  challenge  indentified  in  the  MTWs 

Analysis  for  each  augmented  mission  thread 

•  Start  with  SoS  Architect 

•  Walkthrough  the  architecture  describing  how  the  architecture 
satisfies  the  MT 

—  Step  by  step  probing  all  highlighted  QAs,  looking  for  risks 
—  Some  hybrid  of  completing  a  step  for  all  QAs  and  completing  all  steps  for 
a  QA. 


For  each  analysis  above: 

•  SoS  architect  can  hand  over  to  system  and  s/w  architects  as  needed 

•  The  evaluation  team  probes  for  risks 

•  Scribe  risks,  non-risks  and  issues,  etc  using  the  evaluation  template 
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Stage  2:  Execution  -  3 


Strong  facilitation  to  stay  on  track;  Do  not  go  too  deep  in 
system  architectures,  whatever  is  architecturally  significant 
for  the  MT  at  the  SoS  level. 

Create  “Parking  Lot”  for  non-technical  issues 

Summarize  findings  in  an  out-brief 
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Stage  3:  Roll-up  and  Follow-up 


At  the  end  of  each  SoS  Arch  Eval: 

•  Output  Briefing 

•  SoS  Architectural  Risk  Themes,  Non  risks,  Trade-offs 

•  Any  non-architectural  issues  discovered 

•  One  example  of  an  mission  thread  analysis  with  discovered  SoS 
architectural  risks,  trade-off  points  and  non-risks 

•  Any  problematic  systems  identified  for  future 

•  Identify  “parking  lot”  issues 

•  Summary  Report  of  individual  SoS  Arch  Eval 

•  Detailed  write-ups  on  the  risk  themes,  non-risks,  etc  found  during 
the  evaluation 

•  Summary  of  the  SoS  architecture,  approaches,  guidelines,  etc 

•  Summary  of  the  SoS  business  and  mission  drivers,  quality 
attributes,  summarizing  implications  of  any  mismatches  between 
SoS  and  systems 
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SoS  Arch  Evals  Roll-up 


At  the  end  of  the  series  of  SoS  Arch  Evals 

•  Evaluation  team  meets  to  roll-up  the  findings  from  the  series  of  SoS 
Arch  Evals 

•  Annotated  Summary  Briefing 

•  SoS  Architectural  Risk  Themes  and  Non-risks  (rolled  up) 

•  Any  non-architectural  issues  discovered  (rolled  up) 

•  Identify  problematic  areas  and  schedule  “focused”  architecture 
evaluations  (e.g.  System  &  Software  ATAM) 

•  Recommendations 

•  SoS  Arch  Eval  Summary  Report 

•  Detailed  write-ups  on  the  risk  themes,  non-risks,  etc  found  during 
the  evaluation 

•  Summary  of  the  SoS  architecture,  approaches,  guidelines,  etc 

•  Summary  of  the  SoS  business  and  mission  drivers,  quality 
attributes,  summarizing  implications  of  any  mismatches  between 
SoS  and  systems 

•  Recommended  Next  Steps 
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Summary  and  Next  Steps 
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Next  Steps 


Extensions 

Programmatic,  Acquisition,  Planning  and  Business  Thread 
Workshops 

•  E.g.  Business  Thread  Workshop  -  “Vignette”  replaced  by  “Business 
Context”,  “Mission  Thread”  replaced  by  “End-to-end  Business  Thread” 

SoS  Acquisition  to  be  more  architecture-centric 

•  RFPs,  SOWs,  acquisition  strategies,  etc 

SoS  Architecture  Guidelines  template 

•  Turn  this  into  a  CDRL 

•  Transition  from  architectural  challenges  to  actionable  items  and 
guideline  development. 
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Contact  Information 


Mike  Gagliardi  -  miq@sei.cnnu.edu 

Software  Engineering  Institute 
Carnegie  Mellon  University 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213 
412-268-7738 


Bill  Wood  -  wqw@sei.cmu.edu 
Tim  Morrow  -  tbm@sei.cmu.edu 
John  Klein  -  jklein@sei.cmu.edu 
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